Implementing sideband control structure for PCIE cable cards and IO expansion enclosures

ABSTRACT

A method, system and computer program product are provided for implementing sideband control structure for Peripheral Component Interconnect Express (PCIE) add-in cards, or cable cards, that utilize cables to connect to input/output (IO) expansion enclosures in a computer system. System firmware uniquely identifies a cable card present in a PCIE slot in a system unit. Enclosure management functions utilize sideband control paths integrated within at least cable providing sideband control signaling, and providing PCIE signaling lanes between the cable card and the IO enclosure.

This application is a continuation application of Ser. No. 14/243,999filed Apr. 3, 2014.

FIELD OF THE INVENTION

The present invention relates generally to the data processing field,and more particularly, relates to a method, system and computer programproduct for implementing sideband control structure for PeripheralComponent Interconnect Express (PCI-Express or PCIE) cable cards andinput/output (IO) expansion enclosures in a computer system.

DESCRIPTION OF THE RELATED ART

Peripheral Component Interconnect Express (PCIE) has become the industrystandard IO bus for server computer systems, as well as personalcomputers (PCs). Traditionally, servers install PCIE IO adapters (IOAs)in slots within a system unit that connect through a PCI host bridge tothe system memory and processor buses. IBM POWER and Z series systemshave offered external IO enclosures to provide additional PCIE slotsbeyond those that are available within the system unit. These have inthe past been connected to the system unit through IBM proprietaryinterconnect architectures such as HSL and Infiniband 12× IO loops onIBM POWER systems.

A need exists to provide an external IO expansion enclosure utilizingPCIE slots in a system unit to connect via one or more cables to PCIEslots in the external IO expansion enclosure that provides additionalPCIE slots, where the cable between the PCIE card in the system unit andan IO module within the external IO enclosure provides, for example, 16lanes of PCI-Express bus, used for normal, standard PCIE configurationand IO transfer operations. A need exists to provide a system unit and aprogrammed management controller (a Chassis Management Controller, orCMC) for providing functions of detecting cable types and connectiontopologies to the IO enclosure; controlling power off/on of the IOenclosure, or components within the drawer, via system software;detecting and reporting exception or error conditions within theenclosure, determining component types and manufacturing information forcomponents within the drawer, and the ability to download and updatefirmware in that controller or CMC. Consequently, an additional sidebandcommunications mechanism is needed, outside of PCIE, between firmware orsoftware running in the system unit and the management controller orother hardware within the enclosure.

SUMMARY OF THE INVENTION

Principal aspects of the present invention are to provide a method,system and computer program product for implementing sideband controlstructure for Peripheral Component Interconnect Express (PCIE) add-incards, or cable cards, that utilize cables to connect to input/output(IO) expansion enclosures in a computer system. Other important aspectsof the present invention are to provide such method, system and computerprogram product substantially without negative effects and that overcomemany of the disadvantages of prior art arrangements.

In brief, a method, system and computer program product are provided forimplementing sideband control structure for Peripheral ComponentInterconnect Express (PCIE) cable cards and input/output (IO) expansionenclosures in a computer system. System firmware uniquely identifies acable card present in a PCIE slot in a system unit. Enclosure managementfunctions utilize sideband control paths integrated within at leastcable providing sideband control signaling, and providing PCIE signalinglanes between the cable card and the IO enclosure.

In accordance with features of the invention, a pair of cablesadvantageously provides redundancy for both the PCIE link and thesideband control path, in the event of failures of one cable.

In accordance with features of the invention, system firmware configuresthe PCIE slot when first initializing the PCIE slot in the system unit,both at system initial program load (IPL) time and at PCIE hot plugpower on.

In accordance with features of the invention, the cable card includescontrol and status registers accessible to firmware in the system unit,and a wire engine and data engine, the wire engine transmits control andstatus signals over the cables to both a corresponding wire engine anddata engine in the IO enclosure, and the data engine transmits commandand response messages and data for management communications between thesystem firmware and a management controller firmware within the IOenclosure. The wire engine and data engine in the IO enclosure receivescontrol signals from and provides status to the system unit wire engine,and receives command messages and data and transmits response messagesand data to the system unit data engine.

In accordance with features of the invention, the control and statussignals implemented in the wire engine and data engine and exchangedbetween the system unit and IO enclosure for purposes of determining orestablishing enclosure states including status by which system firmwarecan determine if either cable is present on the system cable card, andto determine whether either cable on the system cable card is connectedto the enclosure; status by which system firmware can determine whichcable position each of the pair of cables from the cable card connectsto on the IO enclosure, enabling firmware to detect incorrectcross-cabling between the cable card ports and ports on the IOenclosure; status by which system firmware can determine which type ofIO module the cables connect to including a single slot within a module,referenced as a Direct Slot module, or a module that expands through aPCIE switch to multiple slots referenced as a Fan-out Module; hardwarecontrol signals from the cable card to the IO enclosure to automaticallyinitiate power on to full power when the cable card has full power, andsimilarly to power off the enclosure when the system cable card powersoff; and control signals that facilitate system firmware transferringcommands, data, and responses utilizing the industry standard I2C bus,and transferring this data across the cables transparently to systemfirmware utilizing the data engine.

In accordance with features of the invention, system firmware detectscable cards present in PCIE slots within the system unit as part ofsystem boot and with PCIE hot plug power on detection and initializationof PCIE buses.

In accordance with features of the invention, system and managementcontroller firmware utilize the sideband control structure to performenclosure management operations and to utilize the redundant pathsprovided with a pair of cables.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention together with the above and other objects andadvantages may best be understood from the following detaileddescription of the preferred embodiments of the invention illustrated inthe drawings, wherein:

FIG. 1 illustrates an example computer system for implementing sidebandcontrol structure for Peripheral Component Interconnect Express (PCIe orPCI-Express) PCIe cable cards and input/output (IO) expansion enclosuresin accordance with a preferred embodiment;

FIGS. 2A, and 2B illustrates novel cable card and sideband controls ofthe example system of FIG. 1 in accordance with a preferred embodiment;

FIGS. 3A, and 3B illustrates further details of novel Local and RemoteFPGA Controllers of the example system of FIG. 1 in accordance with apreferred embodiment;

FIGS. 4A, and 4B together illustrates example operational features ofwire engine registers at host side wires at IO drawer end of the examplesystem of FIG. 1 in accordance with a preferred embodiment;

FIG. 5 illustrates example operational features of a physical layer viewof I2C multiplexer (MUX) and data engine at host side and IO drawer endof the example system of FIG. 1 in accordance with a preferredembodiment;

FIG. 6A is a flow chart illustrating example operational features oflink start up of the example system of FIG. 1 in accordance with apreferred embodiment;

FIG. 6B is a flow chart illustrating example link operational featuresof the example system of FIG. 1 in accordance with a preferredembodiment;

FIG. 7 is a flow chart illustrating example firmware operationalfeatures using sideband status information of the example system of FIG.1 in accordance with a preferred embodiment; and

FIG. 8 is a block diagram illustrating a computer program product inaccordance with the preferred embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following detailed description of embodiments of the invention,reference is made to the accompanying drawings, which illustrate exampleembodiments by which the invention may be practiced. It is to beunderstood that other embodiments may be utilized and structural changesmay be made without departing from the scope of the invention.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

In accordance with features of the invention, a method, system andcomputer program product implement sideband control structure forPeripheral Component Interconnect Express (PCIE or PCI-Express) PCIEcable cards and input/output (IO) expansion enclosures in a computersystem.

Having reference now to the drawings, in FIG. 1, there is shown anexample computer system generally designated by the reference character100 for implementing sideband control structure for Peripheral ComponentInterconnect Express (PCIE or PCI-Express) PCIE cable cards andinput/output (IO) expansion enclosures in accordance with a preferredembodiment. Computer system 100 includes a computer Central ElectronicsComplex (CEC) 102 that includes a system motherboard 104 common in theart that provides for the physical packaging interconnection of thevarious components illustrated within the CEC 102. Computer system 100includes an input/output (IO) expansion enclosure or IO drawer generallydesignated by the reference character 106 in accordance with a preferredembodiment.

In accordance with features of the invention, the IO drawer 106 is anelectrical enclosure that provides PCIE add-in card slots (PCIE slots)or integrated devices utilized by firmware and software operating withinthe CEC 102 to extend the number of PCIE devices physically configurablewithin the scope of the CEC.

CEC system motherboard 104 includes one or more processors or centralprocessor units (CPUs), such as processor 110. Processor 110 is suitablyarranged for executing firmware and software, such as operating systemand applications hosted by respective operating systems in control ofthe overall computer system 100 in accordance with a preferredembodiment.

CEC system motherboard 104 includes one or a plurality of PCIE rootcomplexes 120. Each PCIE root complex 120 includes one or a plurality ofPCIE host bridges (PHBs) 130, each PHB 130 providing a respective PCIElink 132 to a respective PCIE slot 134 mounted commonly on the systemmotherboard 104. For example, the PCIE link 132 is comprised of aplurality of 16 lanes of PCIE signaling wires and auxiliary signals,such as specified in the PCIE standard. The auxiliary signals includePCIE PERST, also called Fundamental Reset, and PCIE clocks from the PHBto a PCIE device, PCIE card present from a device to a PHB, and othersuch discrete signals that operate independently of the PCIE signalinglanes.

One or more of the PCIE slots 134 includes a respective PCIE cable card140 including a respective PCIE cable connector 142 connected by a pairof PCIE cables 144 to a corresponding respective PCIE cable connector142 provided with the IO drawer 106, as shown.

IO drawer 106 includes one or a plurality of module bays, such as adirect slot bay 150 including a plurality of PCIE slots 152, and afan-out bay 154, including a PCIE switch 156 connected to anotherplurality of PCIE slots 152, as shown. Each of the direct slot bay 150,PCIE slots 152, and the fan-out bay 154 is connected to the CEC 102 bythe individual and independent cables 144 via PCIE cable connectors 142,as shown. The PCIE switch 156 includes an upstream port 158 connectedupstream to the PCIE Cable Connector 142 and downstream to a pluralityof downstream ports 160, each connected to a respective PCIE slot 152,as shown.

Computer system 100 is shown in simplified form sufficient forunderstanding the present invention. The illustrated computer system 100is not intended to imply architectural or functional limitations. Thepresent invention can be used with various hardware implementations andsystems and various other internal hardware devices.

In accordance with features of the invention, one or a plurality ofcables 144 is provided with each cable card 140 and respective PCIE slot152 or PCIE switch 156. As shown, a pair of PCIE cables 144 convey PCIEsignal lanes and auxiliary signals forming a single PCIE link, and thePCIE cables 144 also convey sideband control and status signals betweenthe CEC 102 and the IO drawer 106. The use of two cables 144 for eachsingle PCIE link between the CEC 102 and IO drawer 106 advantageouslyenables redundancy in the event of a cable failure. Only a single cable144 is utilized in communicating sideband controls and status betweenthe CEC and IO drawer, and each of the two cables 144 provides analternative signaling path to communicate such sideband controls.

In accordance with features of the invention, the cables 144 may beconventional copper cables or fiber optic cables employingopto-electronic transceivers at each cable end. For example, withrelatively short distances, such as inches to a few feet, copper cables144 are generally suitable for conveying PCIE lanes operating atrequired PCIE signaling rates between the CEC 102 and IO drawer 106.Optical cables 144 advantageously provide electrical isolation to enablehigh signaling rates over longer distances, such as one to many meters.

In accordance with features of the invention, use of optical PCIE cables144 includes serializing required DC signals over the optical cables 144and then converting the serialized signals back to DC signals at theother end of the optical PCIE cables 144 at CEC 102 or the IO drawer106.

In accordance with features of the invention, firmware operating in theCEC 102 is enabled to determine a type of cable card 140 plugged into aPCIE slot 134, with a plurality of different such cable card typeshaving differing properties, such as whether the cables 144 are copperor optical, how many cables 144 emanate from the cable card 140, and thelike. It is similarly advantageous for firmware operating in the CEC 102to be able to determine whether the cables 144 are actually connected toan IO drawer 106, and what type of IO drawer 106, and the IO module bay150, or module bay 156, is connected to cables 144.

In accordance with features of the invention, in a logically partitionedcomputer system 100 firmware operating in the CEC 102 is enabled toassign respective CEC PCIE slots 134 to a particular logical partitionwhen that slot 140 contains a PCIE IO adapter or firmware operating inthe CEC 102 is enabled to detect a cable card 140 and to assignrespective CEC PCIE slots 152 to different logical partitions with thecable card 140 connected to an IO drawer 106.

Referring also to FIGS. 2A, and 2B, additional example details of novelcable card 140 and sideband controls generally designated by thereference character 201 are shown of computer system 100 of FIG. 1 inaccordance with a preferred embodiment. As shown in FIG. 2A, includedwithin or accessible to the processor 110 is an Inter-Integrated Circuit(I2C) master device 202, which is the master of an I2C bus 204 utilizedas an IO bus of the sideband signaling apparatus of a preferredembodiment. It should be understood that other such IO buses known inthe art may be suitable to substitute for the I2C bus 204 utilized bythe invention. Within CEC 102, the I2C bus 204 is connected between theI2C master device 202 and a card present port expander 206, PCIE slots134, a vital product data (VPD) chip 208, and a local control fieldprogrammable gate array (FPGA) 210 provided within the cable card 140.

The PCIE cable card 140 utilizes pins within the PCIE connector 142 ofthe PCIE slot 134 defined in PCIE as reserved pins to generate a signalidentifying the PCIE cable card 140 as a cable card. The card presentport expander 206 connected on the I2C bus 204 receives a card presentsignal from the cable card 140 uniquely indicating the presence of acable card, as opposed to a PCIE IO adapter. Firmware operating in theCEC 102 utilizes the I2C master 202 to read registers within the cardpresent port expander 206 in order to determine that the cable card 140is plugged in the respective PCIE card slot 134. It should be understoodthat other devices than the card present port expander 206 could be usedto receive cable card present information in a manner accessible tofirmware operating within the CEC 102.

The local control FPGA 210 includes registers that receive status fromand optionally signal controls to other hardware components located onthe cable card 140. The registers within the local control FPGA 210 areconnected to the I2C bus 204 proceeding from the PCIE slot 140 onto thecable card 140.

Referring also to FIG. 2B, the IO drawer 106 similarly includes a remotecontrol FPGA 250. The remote control FPGA 250 includes registers thatreceive signals from other hardware components internal to the IO drawer106. The IO drawer 106 includes a drawer controller 252 coupled to theremote control FPGA 250 via an I2C bus 254.

In accordance with features of the invention, as shown in FIGS. 2A, and2B, sideband controls 201 are coupled between the local control FPGA 210in the CEC 102 and the remote control FPGA 250 in IO drawer 106 by a lowbyte cable 260 and a high byte cable 262. For example, the low bytecable 260 conveys PCIE lanes 0 through 7 of the PCIE link from the PCIEslot 134, shown as Low Byte PCIE 264, and conveys sideband signalsbetween the cable card 140 and the IO drawer 106, shown as Low ByteControl 266. For example, the high byte cable 262 conveys PCIE lanes 8through 15 of the PCIE link from the PCIE slot 134, shown as High BytePCIE 268, and conveys sideband signals between the cable card 140 andthe IO drawer 106, shown as High Byte Control 270. For example, the highbyte cable 262 serves as an alternate or redundant connection to the lowbyte cable 260 for the purpose of conveying sideband signals.

Using either the low byte control 266 or high byte control 270, theremote control FPGA 250 signals changes in the states of varioushardware components or DC voltage signals within the IO drawer 106 tothe local control FPGA 210, which receives these changes in registersaccessible to firmware operating in the CEC 102. Similarly, firmwareoperating in the CEC 102 may set register values in the local controlFPGA 210 directed at the remote control FPGA 250 to change the state ofhardware components or DC voltage signals within the IO drawer 106.

Using the either the low byte control 266 or high byte control 270, thelocal FPGA 210 communicates local FPGA 210 register changes to theremote control FPGA 250. The registers within the remote control FPGA250 connect to the I2C bus 254 within the IO drawer 106. The remote FPGAregisters are also accessible as I2C devices from the local control FPGA210. Firmware operating in the CEC 102 utilizes registers in the localcontrol FPGA 210 to create I2C bus operations transmitted between thelocal control FPGA 210 and remote control FPGA 250 utilizing the lowbyte control 266 or high byte control 270. The local control FPGA 210enables firmware operating within the CEC 102 to determine variousconfiguration and operational states of hardware components or DCvoltage signals located on the cable card 210 as well as hardwarecomponents or DC voltage signals within the IO drawer 106.

The drawer controller 252 connected to the remote control FPGA 250within the IO drawer 106 monitors or manages states of the hardwareinternal to the IO drawer, such as turning on or off power supplieswithin the drawer, monitoring thermal or electrical states of componentswithin the drawer, taking actions in response to particular thermal orelectrical states or thresholds, and the like. The drawer controller 252connects to the remote control FPGA 250 utilizing the I2C bus 370,enabling the drawer controller 252 to read or write registers within theremote control FPGA 250 and to communicate status to or receive controlinformation communicated from the local control FPGA 210 using the lowbyte control 266 or high byte control 270.

Referring also to FIGS. 3A, and 3B, additional example details of novellocal control and remote control FPGAs 210, 250 are shown of the examplesystem 100 of FIG. 1 in accordance with a preferred embodiment.

The cable card 140 is shown connected to the IOA drawer 106 utilizing alow byte cable 302 and a high byte cable 304. The low byte cable 302conveys the low byte PCIE signals 306 representing PCIE lanes 0 to 7 andcontrol signals between the local FPGA 210 and remote FPGA 250 indicatedas low byte control 308. The high byte cable 304 conveys the high bytePCIE signals 310 representing PCIE lanes 8 to 15 and control signalsbetween the local FPGA 210 and remote FPGA 310 indicated as high bytecontrol 312. The signals conveyed by means of the low byte control 308and high byte control 312 signals may be communicated over either orboth of the low byte cable 302 and the high byte cable 304 at any onetime, such that each cable can convey the control signals as a backupfor the other in the event of failure or disconnection of one cable 302or 304, and such that signals may be communicated over both cables inorder to detect the location to which each cable is connected at the IOdrawer 106.

The low byte control 308 and low byte PCIE 306 signals in the low bytecable 302, and the high byte control 312 and high byte PCIE 310 signalsin the high byte cable 304 are conveyed optically utilizing a respectiveoptical transceiver (XCVR) 318 on the cable card 140 and opticaltransceiver (XCVR) 358 in the IO drawer 106. The PCIE lanes 0 to 7conveyed on the low byte PCIE 306 and lanes 8 to 15 conveyed on the highbyte PCIE 310 commonly pass through a respective PCIE re-timer 320, 360in order to synchronize them with the respective optical transceivers318, 358.

The local control FPGA 210 on the cable card 140 includes a data engine322, registers 324, a link engine 326, and a wire engine 328. The remotecontrol FPGA 250 similarly includes a data engine 362, registers 364, alink engine 366, and a wire engine 368, and optionally a flash memory270 coupled to the data engine 362. The local control FPGA data engine210 and remote control FPGA data engine 250 are capable of exchangingcontrol signals utilizing either the low byte control 308 or high bytecontrol 312 conveyed over the low byte cable 302 or high byte cable 304,respectively. The link engine 326 utilizes the data engine 322 toestablish reliable optical signaling and bit transfer protocols betweenthe optical XCVRs 318 on the cable card 140 and the optical XCVRs 358and data engine 362 in the IO drawer 106 over both of the low byte cable302 and high byte cable 304.

The wire engine 328 of local control FPGA 210 receives the state ofcertain bits of the registers 324 or DC voltage signals and utilizes thedata engine 322 to transmit these states to the registers 364 of theremote control FPGA 250. The registers 324 include a predefined bit toassert the state of the PCIE auxiliary PERST DC voltage signal outputfrom a PHB 130 to a device attached to the respective PCIE link, and abit to receive the state of PCIE auxiliary device present DC voltagesignal from a PCIE slot 152 in the IO drawer 106 connected to the PCIElink over the low byte cable 302 and high byte cable 304. When the stateof certain bits of registers 324 changes, the wire engine 328automatically communicates these to registers 364 of the remote FPGA250. The wire engine 368 of remote control FPGA 250 receives the stateof certain bits of the registers 364 or DC voltage signals and utilizesthe data engine 362 to transmit these states to the registers 324 of thelocal control FPGA 210. Whenever the state of these certain bits ofregisters 364 changes, the wire engine 368 automatically communicatesthese to registers 324 of the local control FPGA 210.

The respective wire engine 328, 368 on each on each end of the opticalcables 302, 304 provide an alternative signaling mechanism for PCIEauxiliary signals or other DC voltage signals with the fiber opticcables to establish or receive the active or inactive state of theauxiliary signals at the respective other end of the cable.

The registers 324 of local control FPGA 210 include bits representingvarious properties of the cable card 140, such as the type of the cablecard itself, the type and connection states of the low byte cable 302and high byte cable 304. The registers 324 include bits to detect thestates of certain hardware inputs from or control the states of certainhardware outputs to the components of the cable card 140. The registers324 of local control FPGA 210 include bits representing variousproperties of the cable connections to that IO drawer, such asrepresenting which location on the IO drawer 106 of the low byte cable302 and high byte cable 304 are connected, to enable firmware todetermine that cables are properly connected.

The cable card 140 and the IO drawer 106 optionally includes Link ActiveLEDs 380 in association with each of the low byte cable 302 and highbyte cable 304. Firmware operating in the CEC 102 utilizes bits withinthe registers 324 of the local control FPGA 210 to active or deactivatethe link active LEDs 380 to indicate that the cable is or is notactively transmitting signals between the cable card 140 and IO drawer106. Firmware operating in the CEC 102 performs other control andcommunications operations, such as activating or deactivating power tothe IO drawer 106, a module 150, or 145, PCIE slots 152, or othercomponents within the IO drawer 106.

FIGS. 4A, and 4B together illustrate example sideband controloperational features at host side FPGA 210 and the IO drawer FPGA 250 ofthe example system 100 in accordance with a preferred embodiment. InFIG. 4A, example sideband control operational features generallydesignated by the reference character 400 that include wire engineregisters 402 at host side FPGA 210 with example DC voltage controlsignals at host side FPGA 210 and the IO drawer FPGA 250. A low lane 404and high lane 406 connect the host side FPGA 210 and the IO drawer FPGA250. For example, the wire engine is responsible for getting data atlocal I2C registers at the host side FPGA 210 to output wires on the IOdrawer side FPGA 250 as well as input wires on the IO drawer side to I2Cregister that can be read at the host side, and this engine runsautomatically. Any time there is a change to wires or registers on oneside, the changed data will immediately be sent to the other side. Theside that receives new data will also sent its data back to the otherend. For example, 16 bit status and location data is exchanged betweenthe two FPGA 210, 250, and this data also is responsible for gettingPERST and PRESENT exchanged.

In FIG. 4B, there are shown example register bits generally designatedby the reference character 410. The example register bits 410 includeexample local FPGA register bits and example remote FPGA register bitsthat are shown on the left side with corresponding from/to functionsshown on right side in FIG. 4B.

Referring to FIG. 5, there are shown example operational featuresgenerally designated by the reference character 500 of a physical layerview of the example system 100 of FIG. 1 in accordance with a preferredembodiment. An I2C multiplexer (MUX) 502 and a respective data engine504, 514 and an I2C SPI flash 510, 520 at host side FPGA 210 and IOdrawer side FPGA 250 are shown connected by CXP lanes. A respective SPI512, 522 is coupled to the I2C SPI flash 510, 520 at host side FPGA 210and IO drawer side FPGA 250. The I2C multiplexer (MUX) 502 and therespective data engine 504, 514 enable access secondary I2C buses in aconsistent manner. The secondary I2C bus 254 could be in the IO drawer106 talking to the drawer controller 252 or it could be the I2C bus 204to each of the cable connector.

Referring to FIG. 6A, there is shown a flow chart illustrating exampleoperational features of link start up of the example system 100 inaccordance with a preferred embodiment. In FIG. 6A, example operationsof local FPGA on the host side or CEC side are shown on the left andexample operations of remote FPGA on the IO drawer side are shown on theright.

As indicated in a block 600, the local FPGA receives power on from theCEC main power. As indicated in a block 601, the remote FPGA is onstandby power, with PERST asserted to PCIE slot or PCIE switch. Thelocal FPGA link engine transmits sync characters on the low byte cablecontrol as indicated in a block 602. The remote FPGA link engine lowbyte control PLL locks on sync as indicated in a block 604. The remoteFPGA link engine transmits sync on low byte control as indicated in ablock 606. The local FPGA link engine transmits syncs on the high bytecable control as indicated in a block 608. The remote FPGA link enginehigh byte control PLL locks on sync from the local FPGA as indicated ina block 610. The local FPGA link engine low byte control PLL locks onsync from the remote FPGA at block 606 as indicated in a block 612. Theremote FPGA link engine transmits sync on high byte control as indicatedin a block 614. As indicated in a block 616, power on of the IO draweris provided responsive to the remote FPGA link engine low byte controlPLL locks on sync at block 604 or the remote FPGA link engine high bytecontrol PLL locks on sync from the local FPGA at block 610. After localFPGA link engine low byte control PLL locks on sync from the remote FPGAat block 612, the local FPGA wire engine sends 24 bit wire data packetto remote FPGA on low byte control as indicated in a block 618.

The local FPGA link engine PLL locks on sync from the remote FPGA highbyte control transmitted at block 614 as indicated in a block 622. Thelocal FPGA wire engine sends 24 bit wire data packet to remote FPGA onhigh byte control as indicated in a block 628. As indicated in a block624, the remote FPGA data engine receives the 24 bit wire data packetsent from the local FPGA wire engine at block 618. As indicated in ablock 626, the remote FPGA data engine sends the 24 bit wire data packeton low byte control to the local FPGA wire engine. As indicated in ablock 630, the local FPGA wire engine receives the 24 bit wire datapacket from the remote FPGA low byte control. As indicated in a block632, the low byte control link is working. As indicated in a block 634,the remote FPGA data engine receives the 24 bit wire data packet on highbyte control from the local FPGA wire engine. As indicated in a block636, the remote FPGA data engine sends the 24 bit wire data packet onhigh byte control to the local FPGA wire engine. As indicated in a block638, the local FPGA wire engine receives the 24 bit wire data packetfrom the remote FPGA on high byte control. As indicated in a block 640,the high byte control link is working.

Referring to FIG. 6B, there is shown a flow chart illustrating examplelink operational features of the example system 100 in accordance with apreferred embodiment. As indicated in a block 650, when link and hoseare trained such as shown in FIG. 6A, the IO drawer FPGA defaults todriving PERST to the PCIE link until it gets a wire data packet thatsays to quit driving PERST. If the IO drawer lost PLL lock on bothlinks, the IO drawer goes again to power on state, quits transmittingsyncs and drives PERST at bock 650. When host lost PLL sync on one ofits receivers, link trained status bit is cleared for that link at block650.

As indicated in a block 652, receiving a negative acknowledge (NACK) oneither side, causes the link status bit to be cleared and packet isresent. With a wire data packet, full exchange is provided before thelink is trained. With an I2C data packet, the link goes to trained whenacknowledge (ACK) is received.

As indicated in a block 654, the IO drawer FPGA does not send anypackets until a wire data packet is received from the host FPGA. Afterthe IO drawer FPGA has responded with its first wire data packet, the IOdrawer FPGA will send a wire data packet any time one of its wire inputchange, and receives an ACK from the host.

As indicated in a block 656, a positive ACK of wire data for host codewhen PERST changes level or there is an I2C write to any bits that arein the wire data packet, the host sends a wire data packet and clear itslink trained bit. When the IO drawer receives a good wire packet italways responds with an ACK and its wire data packet.

As indicated in a block 658, when the host receives the wire data packetfrom the IO drawer and has sent an ACK, the host sets its link trainedbit.

Referring to FIG. 7, there is shown a flow chart illustrating examplefirmware operational features using sideband status information of theexample system 100 in accordance with a preferred embodiment.

As indicated in a decision block 700, checking for a PCIE card presentin CEC PCIE slot is performed. When a PCIE card is present in CEC PCIEslot, the cable card present port expander is read as indicated in ablock 702. As indicated in a decision block 704, again checking for thePCIE cable card present in CEC PCIE slot is performed.

When a PCIE cable card is present in CEC PCIE slot, the local FPGAregisters are read to get cable status and connection locations asindicated in a block 706. Checking if the cable card includes a pair ofcables or dual cables is performed as indicated in a decision block 708.When the cable card includes dual cables, checking is performed todetermine if the high byte control is working as indicated in a decisionblock 710. If the high byte control is not working, a cable error islogged for service as indicated in a block 712. If the high byte controlis working and when the cable card does not include dual cables,checking is performed to determine if the low byte control is working asindicated in a decision block 714. If the low byte control is notworking, a cable error is logged for service as indicated in a block716.

As indicated in a block 718, local FPGA registers are read to verifycables are correctly connected low to low, high to high, and same PCIElink connection at the IO drawer. Checking if cabled correctly isperformed as indicated in a decision block 720. If not cabled correctly,a cable error is logged for service as indicated in a block 722. Ifcabled correctly, the local FPGA registers are written to de-assertPERST to the IO drawer PCIE link connection as indicated in a block 724.Checking is performed to determine if connected to PCIE switch in IOdrawer as indicated in a decision block 726. When connected to PCIEswitch in IO drawer, the PCIE switch and downstream PCIE links to PCIEslots in the IO drawer are configured as indicated in a block 728. ThePCIE slot or PCIE slots in the IO drawer under the PHB connected to thiscable are configured as indicated in a block 730. The PCIE linkconnection configuration is complete as indicated in a block 732.

Referring now to FIG. 8, an article of manufacture or a computer programproduct 800 of the invention is illustrated. The computer programproduct 800 is tangibly embodied on a non-transitory computer readablestorage medium that includes a recording medium 802, such as, a floppydisk, a high capacity read only memory in the form of an optically readcompact disk or CD-ROM, a tape, or another similar computer programproduct. Recording medium 802 stores program means 804, 806, 808, and810 on the medium 802 for carrying out the methods for implementingsideband control structure for PCIE cable cards 140 and IO expansionenclosures 106 of preferred embodiments in the system 100 of FIG. 1.

A sequence of program instructions or a logical assembly of one or moreinterrelated modules defined by the recorded program means 808, 806,808, and 810, direct the computer system 100 for implementing sidebandcontrol structure for PCIE cable cards 140 and IO expansion enclosures106 of a preferred embodiment.

While the present invention has been described with reference to thedetails of the embodiments of the invention shown in the drawing, thesedetails are not intended to limit the scope of the invention as claimedin the appended claims.

What is claimed is:
 1. A method for implementing sideband controlstructure for Peripheral Component Interconnect Express (PCIE) cablecards and input/output (IO) expansion enclosures in a computer system,comprising: providing a Central Electronics Complex (CEC) including aprocessor and system firmware for uniquely identifying a cable cardpresent in a PCIE slot in a system unit; providing the cable cardincluding control and status registers accessible to said systemfirmware, a system unit link engine, a system unit wire engine and asystem unit data engine; providing an IO enclosure includingcorresponding control and status registers, a corresponding link engine,a corresponding wire engine and a corresponding data engine; providing arespective first PCIE cable connector connected to a respective secondPCIE cable connector by a respective pair of PCIE cables connectingbetween the cable card and the IO enclosure; providing sideband controlsignaling for implementing enclosure and PCIE link management functionsutilizing sideband control paths integrated within the respective pairof PCIE cables, and providing PCIE signaling lanes within the respectivepair of PCIE cables and each cable of the respective pair of PCIEproviding an alternative signaling path for communicating sidebandcontrols; said system unit wire engine, transmitting control and statussignals over the PCIE cables to both said corresponding wire engine anddata engine in the IO enclosure, and said system unit data engine,transmitting command and response messages and data for managementcommunications between said system firmware and a management controllerfirmware within the IO enclosure; and said wire engine and data enginein the IO enclosure, receiving control signals from and providing statusto said system unit wire engine, and receiving command messages and dataand transmitting response messages and data to said system unit dataengine.
 2. The method as recited in claim 1, includes providing therespective pair of PCIE cables connecting between the cable card and theIO enclosure, enabling redundancy for both the PCIE link and thesideband control path, in the event of failure of one PCIE cable.
 3. Themethod as recited in claim 1, includes implementing said control andstatus registers, said link engine, wire engine and data engine with alocal control field programmable gate array (FPGA) in the cable card,and implementing said registers, said corresponding link engine, wireengine and data engine with a remote control FPGA in the IO enclosure.4. The method as recited in claim 1, includes the link enginetransmitting sync characters over the cables to the corresponding linkengine in the 10 enclosure for establishing communication signaling overthe cables.
 5. The method as recited in claim 4, includes the dataengine transmitting command and response messages and data formanagement communications between the system firmware and a managementcontroller firmware within the IO enclosure.
 6. The method as recited inclaim 5, includes the wire engine and data engine in the IO enclosurereceiving control signals from and providing status to the system unitwire engine, and receiving command messages and data and transmittingresponse messages and data to the system unit data engine.
 7. The methodas recited in claim 1, includes the wire engine and data engineexchanging control and status signals between the system unit and IOenclosure for establishing enclosure states.
 8. The method as recited inclaim 7, includes providing control signals from the cable card to theIO enclosure to automatically initiate power on, and to power off theenclosure.
 9. The method as recited in claim 7, includes providingcontrol signals from the cable card to the IO enclosure to identifyrespective pair of PCIE cables connecting between the cable card and theIO enclosure, to train one or more PCIE links and transfer data acrossthe one or more PCIE links transparently to system firmware utilizingthe data engine.
 10. The method as recited in claim 1, wherein providingsystem firmware for uniquely identifying a cable card present in a PCIEslot in a system unit includes system firmware provides cable carddetection with system boot and with PCIE hot plug power on detection andinitialization of PCIE links.